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ABSTRACT 

An  evaluation  is  presented  on  the  performance  of  the  detection  and  tracking 
algorithms  used  in  a  DSTO  developed  experimental  air  to  air  threat  warner 
(Cyclops).  The  simulations  of  the  Cyclops  routines  were  performed  using  the  Generic 
Track  Processor  software.  It  was  found  that  human  observers  with  limited  training, 
who  were  aware  of  the  approximate  target  location,  reliably  detected  the  target  at 
ranges  significantly  greater  than  the  automatic  target  acquisition  system.  The 
limitations  of  the  data  set  and  implications  of  the  results  are  discussed. 
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Evaluation  of  Cyclops  Automatic  Target  Detection 
and  Comparison  with  Human  Capabilities. 


EXECUTIVE  SUMMARY 


Images  sequences  obtained  during  the  trial  of  Cyclops  (a  DSTO  developed 
experimental  air  to  air  threat  warner)  were  processed  using  a  suite  of  algorithms  - 
referred  to  here  as  the  Generic  Track  Processor,  or  GTP,  to  evaluate  the  Cyclops 
tracking  algorithms.  The  parameters  in  the  Generic  Track  Processor  were  optimised  for 
the  sequences.  The  detection  ranges  were  recorded  and  compared  to  the  detection 
ranges  of  a  trained  operator  and  four  observers  of  a  previous  experiment  who  were  not 
trained  with  the  imagery. 


In  general,  the  trained  operator  detected  the  target  at  least  three  nautical  miles  earlier 
than  the  Generic  Track  Processor  and  two  nautical  miles  earlier  than  the  average 
observer.  This  was  due  to  the  poor  quality  of  the  imagery  which  was  caused  by  sensor 
imperfections  and  made  all  aspects  of  the  tracking  difficult.  However,  when  the 
imagery  was  of  a  good  quality  the  automatic  target  detection  improved.  The  time  spent 
looking  at  the  imagery  also  gave  the  trained  operator  an  advantage  over  the  average 
observer  (the  four  observers  of  the  previous  experiment)  and  the  Generic  Track 
Processor. 


The  tracking  procedures  for  the  Cyclops  data  are  inadequate  due  mainly  to  the  poor 
quality  of  the  images;  however  the  performance  of  the  automatic  target  detection 
process  on  good  quality  images  cannot  be  judged. 


THIS  PAGE  INTENTIONALLY  BLANK 


Contents 


1.  INTRODUCTION . 1 

2.  PROCEDURE . 1 

2.1  Image  Sequence  Selection . 1 

2.2  Processing  Chain . 2 

2.3  Human  Performance . 2 

3.  RESULTS . 3 

4.  ANALYSIS  OF  RESULTS . 4 

5.  CONCLUSIONS . 6 

6.  ACKNOWLEDGMENTS . 7 

7.  REFERENCES . 7 


DISTRIBUTION  LIST 


DOCUMENT  DATA  CONTROL  SHEET 


THIS  PAGE  INTENTIONALLY  BLANK 


DSTO-TN-0033 


1.  Introduction 

Cyclops  was  an  experimental  air  to  air  threat  warning  system  based  on  a  128  by  128 
mercury-cadmium-telluride  focal  plane  array.  The  Cyclops  development  program 
culminated  in  airborne  trials  in  April  1991,  in  which  detection  and  tracking  of  a  distant 
aircraft  approaching  head-on  with  engine  plume  not  visible,  was  investigated. 

The  aim  of  this  experiment  was  to  evaluate  the  performance  of  the  Cyclops  tracking 
algorithms  on  the  Cyclops  data  and  to  compare  these  results  with  those  obtained  by 
human  operators.  The  simulation  of  the  Cyclops  routines  was  performed  using  the 
Generic  Track  Processor  software.  Results  are  presented  and  comparisons  and 
explanations  are  discussed. 


2.  Procedure 

The  procedure  used  to  process  the  Cyclops  data  follows  the  algorithm  as  originally 
proposed  for  the  Cyclops  trial,  but  which  was  not  evaluated  due  to  equipment 
malfunction.  The  results  obtained  here  do  not  duplicate  the  results  that  would  have 
been  obtained  using  the  exact  Cyclops  processing  chain.  The  size  of  filter  windows  and 
other  parameters  have  been  chosen  to  give  the  best  performance. 

2.1  Image  Sequence  Selection 

Eleven  sequences,  each  consisting  of  90  frames  obtained  during  the  Cyclops  trials, 
were  extracted  from  the  Honeywell  HDlOle  magnetic  tape  drive  at  ranges  where  a 
human  operator  could  detect  the  targets.  In  each  sequence,  the  aircraft  was  at  a 
different  location  ( eg.  Tindal  or  Richmond  )  and  at  varying  altitudes  and  speeds.  These 
sequences  were  run  through  the  Generic  Track  Processor  so  that  the  parameters  of  the 
process  could  be  optimised.  None  of  the  targets  was  detected  automatically  in  these 
sequences  and  the  parameters  were  modified  to  allow  for  maximum  sensitivity, 
without  having  an  excessive  amount  of  false  detections.  The  number  of  false  detections 
were  too  high  for  an  operational  threat  warner,  however  the  detections  were  not  so 
numerous  that  they  obscured  a  valid  detection  when  it  occurred. 

Failure  to  detect  the  target  meant  that  extra  image  sequences  from  each  run  had  to  be 
obtained  in  which  the  target  was  closer  to  the  sensor.  Using  this  approach,  the  target 
was  eventually  detected  in  each  run.  The  trained  operator  detection  range,  tracker 
detection  range  and  the  number  of  tracks  in  the  initiated  frame  were  recorded. 
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2.2  Processing  Chain 

The  data  was  first  spatially  filtered  by  means  of  a  low  pass  non-linear  filter  to  remove 
low  frequency  components  from  the  image,  such  as  those  arising  from  illumination 
variation  within  the  image,  or  changing  backgrounds.  A  binary  threshold  was  applied 
to  the  image  to  select  areas  which  are  different  to  the  background.  The  resultant  points 
were  formed  into  clusters.  The  cluster  centres  were  then  either  associated  to  existing 
tracks,  using  a  nearest  neighbour  association  technique,  or  formed  into  new  tracks 
using  an  m-out-of-n  track  initiation  process.  Temporal  filtering  of  the  track  data  used  a 
Kalman  filter. 

A  7x7  pixel  window  was  used  for  the  low  pass  non-linear  filter.  Tests  were  run  on  a 
number  of  different  size  spatial  filter  kernels  ranging  from  3x3  to  9x9.  The  track  was 
initiated  in  the  same  frame  and  the  amount  of  tracks  in  the  initiated  frame  was  about 
the  same  in  all  these  cases  except  for  the  3x3.  In  this  case,  the  track  was  initiated 
slightly  later,  the  number  of  tracks  in  the  initiated  frame  was  lower  and  the  track  was 
not  associated  in  as  many  frames  as  the  larger  kernels  (track  not  at  maximum 
likelihood) .  The  7x7  kernel  was  a  good  mid  range  choice  when  these  results  were  taken 
into  consideration. 

Nearest  neighbour  association  was  performed  in  a  5x5  window.  Larger  association 
windows  were  tried;  however  these  resulted  in  the  track  being  associated  to  the  wrong 
plot.  This  could  be  quite  a  distance  from  the  track  position  and  resulted  in  the  loss  of 
the  real  track  in  the  next  frame. 

The  m-out-of-n  track  initiation  parameters  were  set  to  2-out-of-5.  Although  this 
resulted  in  a  large  number  of  false  tracks,  higher  confidence  initiation  parameters  such 
as  5  out  of  5,  resulted  in  extremely  poor  automatic  detection  ranges  where  the  target 
was  easily  visible  to  a  human  observer. 

The  Kalman  filter  was  detuned  so  that  it  did  not  form  overly  confident  predictions  of 
target  motion.  This  was  necessary  because  of  the  random  manner  in  which  the  target 
would  move  around  in  the  image  due  to  the  vibrations  of  the  platform  mount.  For  this 
report  automatic  detection  is  determined  from  initiation,  and  so  the  parameters  of  the 
Kalman  filter  do  not  affect  these  results.  However,  the  parameters  used  produced  the 
smoothest  possible  track  without  loosing  track  due  to  the  jitter. 

2.3  Human  Performance 

A  previous  experiment  had  been  undertaken  which  involved  a  number  of  observers 
trying  to  detect  the  target  in  image  sequences  from  the  Cyclops  trials.  The  observers 
were  shown  at  least  one  trial  run  to  demonstrate  the  type  of  images  they  would  be 
looking  at,  and  to  familiarise  them  with  the  type  of  motion  and  approximate  location  of 
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the  target  in  the  images.  The  image  sequences  were  started  at  different  lead  times 
before  the  target  was  visible  and  therefore  the  observer  did  not  know  when  to  expect 
the  target.  Each  observer  saw  6  to  8  runs  in  a  row,  which  were  then  repeated  in  the 
same  order.  Information  from  this  experiment  such  as  the  average  detection  range, 
maximum  detection  range  and  minimum  detection  range  was  recorded  where  possible 
(This  experiment  was  not  carried  out  on  sortie  4  and  sortie  7  run  3). 


3.  Results 

The  results  presented  here  list  human  detection  ranges  and  automatic  initiation 
ranges.  The  automatic  target  detection  process  is  considered  to  have  detected  a  track 
when  a  track,  which  will  become  a  high  confidence  track,  is  initiated.  A  human 
observer  will  form  a  level  of  confidence  in  the  track  before  signifying  detection.  The 
different  measures  are  necessary  because  of  the  nature  of  the  imagery,  particularly  the 
manner  in  which  the  lack  of  stabilisation  causes  the  target  to  bounce.  This  makes  it 
difficult  for  the  Kalman  process  to  establish  a  reasonable  level  of  confidence  in  the 
track,  while  human  observers  need  to  establish  confidence  before  they  can  indicate  a 
detection.  Due  to  random  displacements  in  the  image,  the  covariance  of  the  plot 
estimates  from  the  Kalman  filter  remain  high.  Therefore  the  covariance  of  the  estimates 
is  not  a  good  measure  of  the  track  confidence. 

A  track  management  unit  in  the  Generic  Track  Processor  monitors  the  consistency  of 
each  track  by  incrementing  a  track  confidence  parameter  if  association  occurs  and 
reducing  the  confidence  if  no  plots  can  be  associated  to  the  track.  This  confidence 
measure  gradually  grew  for  the  target  in  question.  However,  the  rate  of  growth  or 
decay  of  this  confidence  measure  is  arbitrary,  as  is  a  threshold  value  which  would 
confirm  the  track.  Therefore,  use  of  this  parameter  to  measure  detection  would  modify 
detection  ranges  arbitrarily.  It  should  be  noted  that  there  were  always  objects  with  a 
higher  track  confidence  than  the  real  target  (most  of  which  were  stationary  in  the 
sensor  field,  and  consequently  easily  removed  as  sensor  artefacts,  but  there  were  also 
other  objects  which  could  not  be  so  easily  removed.)  The  different  measures  are 
unfortunate  because  they  reduce  the  apparent  difference  between  the  automatic  target 
detection  process  and  that  of  the  human  observer.  Therefore,  the  automatic  tracker  is 
actually  worse  than  it  appears  from  Table  1  . 

Five  human  observers  viewed  the  sequences.  One  individual,  known  as  the  trained 
operator  whose  detection  ranges  were  not  included  in  the  average  detection  ranges  of 
the  other  observers,  was  allowed  to  view  the  imagery  as  frequently  as  desired.  The 
large  amount  of  time  spent  viewing  the  imagery  meant  the  trained  human  operator 
became  skilled  at  the  early  detection  of  the  target.  The  other  four  individuals,  whose 
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detection  ranges  were  used  to  make  up  the  average  detection  range  by  observers,  were 
presented  the  sequences  in  a  controlled  manner  [2].  They  saw  each  sequence  twice,  but 
not  immediately  after  the  other  run  and  not  necessarily  commencing  at  the  same  point. 
Both  observations  were  recorded  for  each  run  from  each  of  the  observers.  The 
maximum,  minimum  and  average  ranges  of  the  resulting  eight  observations  for  each 
run  are  presented  in  Table  1.  Unfortunately,  the  observations  were  not  conducted  on 
every  run  selected  for  processing.  These  results  are  presented  more  completely  in  [2] 


Table  1:  Table  of  Results 


Image 

Sequence 

Trained 

Operator 

Detection 

Range 

(naut.miles) 

Tracker 

Initiation 

Range 

(naut.miles) 

Average 
Detection 
Range  by 
Observers 
(naut.miles) 

Maximum 
Detection 
Range  by 
Observers 
(naut.miles) 

Minimum 
Detection 
Range  by 
Observers 
(naut.miles) 

Tracks  at 
Initiated 
Frame 

Sortie  8  Run  5 

10.1 

7.4 

9.3 

11.3 

7.2 

120 

Sortie  7  Run  1 

7.0 

2.7 

5.2 

6.1 

3.8 

31 

Sortie  7  Run  2 

10.7 

7.9 

8.8 

9.8 

6.6 

19 

Sortie  7  Run  3 

8.2 

5.1 

Not  obtained 

Not  obtained 

Not  obtained 

33 

Sortie  7  Run  4 

14.2 

8.7 

8.5 

10.7 

7.5 

36 

Sortie  7  Run  5 

5.3 

2.6 

4.6 

5.5 

3.9 

112 

Sortie  4  Run  1 

12.5 

5.9 

Not  obtained 

Not  obtained 

Not  obtained 

18 

Sortie  4  Run  3 

16.9 

11.7 

Not  obtained 

Not  obtained 

Not  obtained 

117 

Sortie  4  Run  4 

17.9 

17.9 

Not  obtained 

Not  obtained 

Not  obtained 

49 

Sortie  4  Run  5 

11.4 

8.6 

Not  obtained 

Not  obtained 

Not  obtained 

32 

Sortie  4  run  6 

11.6 

7.5 

Not  obtained 

Not  obtained 

Not  obtained 

34 

4.  Analysis  of  Results 

The  trained  operator  performed  better  than  the  inexperienced  operators  by  an  average 
of  about  2.2  nautical  miles.  All  human  operators  performed  better  than  the  automatic 
detection  algorithm  with  the  trained  operator  obtaining  detections  of  the  target  at  least 
3  nautical  miles  before  the  tracker  in  all  cases  except  one. 

In  sortie  7  run  4,  the  tracker  performed  better  than  the  average  human  observer.  In 
sortie  7  run  2,  the  tracker's  detection  occurred  shortly  after  the  average  human 
response.  It  should  be  noted  that  only  4  people  were  used  to  obtain  the  average 
detection  response  and  the  results  obtained  on  the  same  run  by  the  same  person  at 
different  times  could  differ  by  3  or  4  nautical  miles. 
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Detection  by  the  tracker  was  hampered  by  the  poor  quality  of  the  imagery.  The  target 
at  times  moved  into  areas  of  sensor  imperfections  like  streaking,  stationary  bright 
pixels  (  dead  pixels  )  and  noise. 
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The  contrast  between  the  target  and  the  background  and  the  poor  sensor  mount 
stability  also  contributed  to  making  early  target  detection  by  the  tracker  difficult. 

The  lack  of  stabilisation  on  the  sensor  mount  caused  the  image  to  appear  to  bounce 
around,  and  made  it  difficult  to  track  and  associate  points  from  successive  frames. 

The  numerous  dead  pixels,  which  remained  stationary  in  the  image  despite  the 
bouncing  around,  are  consequently  easy  to  associate  and  form  high  confidence  tracks. 
The  dead  pixels  also  provide  other  points  for  the  tracker  to  associate  to  the  target  track 
instead  of  the  real  target,  and  this  caused  valid  tracks  to  go  astray.  The  dead  pixels  and 
their  tracks  could  have  been  eliminated  using  additional  processing,  which  relied  on 
the  knowledge  that  false  tracks  due  to  sensor  imperfections  have  a  zero  velocity 
component,  but  this  was  outside  the  scope  of  the  analysis. 

Streaking  caused  by  different  gain  and  offset  parameters  in  the  pixel  read  out 
amplifiers  (each  line  of  pixels  was  read  through  the  same  amplifier)  of  the  sensor 
caused  the  target  intensity  to  fluctuate.  This  also  confused  the  spatial  filtering  front 
end.  This  resulted  in  numerous  false  clusters.  A  simple  de-streaking  algorithm,  which 
removed  the  offset  but  did  not  allow  for  the  variation  in  amplifier  gain,  was  added  in 
an  attempt  to  minimise  the  problems  associated  with  the  streaking.  This  proved  to  be 
moderately  successful  at  removing  many  of  the  false  clusters  generated  during  the 
spatial  filtering. 

All  these  imperfections  hampered  the  preprocessing  of  the  image,  track  association 
and  track  initiation.  An  example  of  the  image  quality  can  be  seen  in  Figure  1. 

The  image  quality  in  sortie  7  run  5  and  sortie  7  run  1  (which  was  described  as  having 
poor  focus)  was  particularly  poor.  In  sortie  4  run  4,  the  operator  detection  range  and 
the  tracker  detection  range  are  identical.  This  is  because  the  image  quality  is  much 
better  in  this  sequence  and  the  bright  target  was  contrasted  against  a  darker 
background.  The  image  sequences  taken  from  sortie  4  non  3  and  sortie  4  run  6  detected 
the  target  earlier  in  the  sequence  but  lost  the  target  before  regaining  it  later  in  that 
sequence. 
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Figure  1.  Example  of  image  quality  at  approximate  time  of  automatic  detection. 


Detection  range  by  the  trained  operator  was  probably  better  than  the  average 
detection  range  due  to  the  experience  of  the  operator  and  knowledge  of  the  expected 
target's  motion,  appearance  and  position  in  the  image.  The  average  observer  would 
have  had  problems  in  detecting  the  target  in  the  early  stages  of  the  sequences  due  to 
the  poor  contrast  of  the  target  with  respect  to  the  background. 


5.  Conclusions 

The  trained  operator  detected  the  target  at  least  3  nautical  miles  earlier  than  the  GTP. 
This  was  due  in  part  to  the  poor  quality  of  most  of  the  imagery.  Sensor  imperfections, 
such  as  image  streaking,  dead  pixels  and  noise,  coupled  with  the  sensor  mount 
instability  made  spatial  filtering  and  all  tracking  aspects  (track  association  and  track 
initiation)  difficult.  The  ability  of  the  human  observer  to  disregard  stationary  points 
and  look  for  motion  of  the  target  within  the  image,  accounts  for  the  superior 
performance  of  human  operators  when  compared  to  the  automatic  detection  process. 

When  good  quality  images  were  available,  the  automatic  tracker  performance 
improved  markedly.  It  is  believed  that  the  tracking  procedures  proposed  for  the 
Cyclops  data  are  inadequate.  This  is  due  mainly  to  the  poor  image  quality.  The 
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performance  of  the  automatic  detection  process  on  good  quality  images  cannot  be 
inferred  from  this  data  set. 
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